Electronic device and control method thereof

ABSTRACT

Disclosed is an electronic device. The electronic device comprises: a microphone comprising circuitry; a speaker comprising circuitry; and a processor electrically connected to the microphone and speaker, wherein the processor, when a first user&#39;s voice is input through the microphone, identifies a user who uttered the first user&#39;s voice and provides a first response sound, which is obtained by inputting the first user&#39;s voice to an artificial intelligence model learned through an artificial intelligence algorithm, through the speaker, and when a second user&#39;s voice is input through the microphone, identifies a user who uttered the second user&#39;s voice, and if the user who uttered the first user&#39;s voice is the same as the user who uttered the second user&#39;s voice, provides a second response sound, which is obtained by inputting the second user&#39;s voice and utterance history information to the artificial intelligence model, through the speaker. In particular, at least some of the methods of providing a response sound to a user&#39;s voice may use an artificial intelligence model learned in accordance with at least one of a machine learning, neural network, or deep learning algorithm.

TECHNICAL FIELD

The disclosure relates to an electronic device and a control methodtherefor, and more particularly to an electronic device which identifiesa user's voice and provides various interactions according to theidentified user's voice and a control method therefor.

The disclosure further relates to an artificial intelligence (AI) systemsimulating functions of the human brain such as recognition ordetermination using a machine learning algorithm, and an applicationthereof.

BACKGROUND ART

Recently, artificial intelligence systems realizing intelligence of ahuman level have been used in various fields. The artificialintelligence system is a system in which a machine trains, determines,and becomes smart, itself, unlike the conventional rule-based smartsystem. As the artificial intelligence system is used, a recognitionrate is improved and preferences of a user can be more accuratelyunderstood, and thus, the conventional rule-based system is graduallybeing replaced with deep learning-based artificial intelligence system.

The artificial intelligence technologies are composed of machinelearning (e.g., deep learning) and elementary technologies using themachine learning. The machine learning is an algorithm technology ofself-classifying/self-training features of input data, and theelementary technology is a technology of simulating functions of thehuman brain such as recognition or determination using the machinelearning algorithm such as the deep learning and is composed oftechnical fields of language understanding, visual understanding,inference/prediction, knowledge representation, operation control, andthe like.

Various fields, to which the artificial intelligence technologies areapplied, are as follows. The language understanding is a technology ofrecognizing languages/alphabets of human and applying/processing it andincludes natural language processing, machine translation, a conversionsystem, question and answer, voice recognition/synthesis, and the like.The visual understanding is a technology of recognizing an object in aview of human and processing it and includes object recognition, objecttracking, image searching, human recognition, scene understanding, spaceunderstanding, image improvement, and the like. The inference/predictionis a technology of identifying the information and logically inferringand predicting it and includes knowledge/possibility-based inference,optimization prediction, preference-based planning, recommendation, andthe like. The knowledge representation is a technology of performingautomating processing of experiment information of human into knowledgedata and includes knowledge construction (datageneration/classification), knowledge management (data application), andthe like. The operation control is a technology of controlling automaticdriving of a vehicle or movement of a robot and includes movementcontrol (navigation, collision, or travelling), manipulation control(behavior control), and the like.

Meanwhile, conventionally, electronic devices which identify a user'svoice and provide various interactions according to the identifieduser's voice have been developed. In addition, the electronic devicesmay not only provide a response to the one-time user's voice, but alsoreceive continuous user's voices and provide responses by reflectingutterance history. However, when a user's voice of another user wasreceived between the continuous the user's voices, the electronicdevices were not able to distinguish these and provided completely wrongresponses.

DISCLOSURE Technical Problem

The disclosure is made in view of the above-mentioned needs and anobject of the disclosure is to provide an electronic device whichprovides a response with a maintained context, although user's voicesare input from a plurality of users, respectively, and a control methodtherefor.

Technical Solution

According to an embodiment of the disclosure for achieving theabove-mentioned object, there is provided an electronic device includinga microphone, a speaker, and a processor configured to be electricallyconnected to the microphone and the speaker, in which the processor isconfigured to, based on a first user's voice being input via themicrophone, identify a user who uttered the first user's voice andprovide a first response sound obtained by inputting the first user'svoice to an artificial intelligence model trained through an artificialintelligence algorithm via the speaker, based on a second user's voicebeing input via the microphone, identify a user who uttered the seconduser's voice, and based on the user who uttered the first user's voicebeing the same as the user who uttered the second user's voice, providea second response sound obtained by inputting the second user's voiceand utterance history information to the artificial intelligence modelvia the speaker.

The device may further include a storage, and the processor may beconfigured to, based on the user who uttered the second user's voicebeing different from the user who uttered the first user's voice, storethe second user's voice in the storage, and based on a user's voice notbeing input from the user who uttered the first user's voice for apredetermined period of time or longer, provide a third response soundobtained by inputting the second user's voice to the artificialintelligence model via the speaker.

The processor may be configured to, based on the first user's voicebeing input, identify a first user who uttered the first user's voiceand provide the first response sound obtained by inputting the firstuser's voice to the artificial intelligence model via the speaker, andbased on the second user's voice being input, identify a second user whouttered the second user's voice and provide a third response soundobtained by inputting the second user's voice to the artificialintelligence model via the speaker to be distinct from the firstresponse sound.

The processor may be configured to, based on a first additional user'svoice being input by the first user, provide a first additional responsesound obtained by inputting the first additional user's voice and firstutterance history information corresponding to the first user's voice tothe artificial intelligence model via the speaker, and based on a secondadditional user's voice being input by the second user, provide a secondadditional response sound obtained by inputting the second additionaluser's voice and second utterance history information corresponding tothe second user's voice to the artificial intelligence model via thespeaker to be distinct from the first additional response sound.

The device may further include a display electrically connected to theprocessor, and the processor may be configured to display a first UIcorresponding to the first user via the display while providing thefirst additional response sound, and display a second UI correspondingto the second user via the display while providing the second additionalresponse sound.

The processor may be configured to, based on the user who uttered thefirst user's voice being different from the user who uttered the seconduser's voice, obtain a first domain corresponding to the first user'svoice and a second domain corresponding to the second user's voice, andbased on the first domain being the same as the second domain, providethe second response sound obtained by inputting the second user's voiceand the utterance history information to the artificial intelligencemodel via the speaker.

The device may further include a storage electrically connected to theprocessor, and the processor may be configured to, based on the seconduser's voice being input within a first predetermined period of timefrom a point when the first user's voice is input, the user who utteredthe first user's voice being the same as the user who uttered the seconduser's voice, and a first domain corresponding to the first user's voicebeing different from a second domain corresponding to the second user'svoice, store the first user's voice in the storage without providing thefirst response sound, and provide a third response sound obtained byinputting the second user's voice to the artificial intelligence modelvia the speaker.

The processor may be configured to, based on a user's voicecorresponding to the second domain not being input from the user for asecond predetermined period of time or longer, provide the firstresponse sound obtained by inputting the first user's voice stored inthe storage to the artificial intelligence model via the speaker.

The device may further include a camera, and the processor may beconfigured to identify the user based on at least one of an imagecaptured by the camera or a user's voice input via the microphone.

The processor may be configured to identify the user who uttered thefirst user's voice based on a shape of mouth of a user included in theimage for a period of time during which the first user's voice is input,and based on the shape of mouth of the identified user included in theimage being changed for a period of time during which the second user'svoice is input, identify the user who uttered the second user's voice tobe the same as the user who uttered the first user's voice.

According to an embodiment of the disclosure, there is provided a methodfor controlling an electronic device, the method including based on afirst user's voice being input, identifying a user who uttered the firstuser's voice, providing a first response sound obtained by inputting thefirst user's voice to an artificial intelligence model trained throughan artificial intelligence algorithm, based on a second user's voicebeing input, identifying a user who uttered the second user's voice, andbased on the user who uttered the first user's voice being the same asthe user who uttered the second user's voice, providing a secondresponse sound obtained by inputting the second user's voice andutterance history information to the artificial intelligence model.

The method may further include based on the user who uttered the seconduser's voice being different from the user who uttered the first user'svoice, storing the second user's voice, and based on a user's voice notbeing input from the user who uttered the first user's voice for apredetermined period of time or longer, providing a third response soundobtained by inputting the second user's voice to the artificialintelligence model.

The identifying a user who uttered the first user's voice may includebased on the first user's voice being input, identifying a first userwho uttered the first user's voice, the providing a first response soundmay include providing the first response sound obtained by inputting thefirst user's voice to the artificial intelligence model, the identifyinga user who uttered the second user's voice may include identifying asecond user who uttered the second user's voice, and the method mayfurther include providing a third response sound obtained by inputtingthe second user's voice to the artificial intelligence model to bedistinct from the first response sound.

The method may further include based on a first additional user's voicebeing input by the first user, providing a first additional responsesound obtained by inputting the first additional user's voice and firstutterance history information corresponding to the first user's voice tothe artificial intelligence model, and based on a second additionaluser's voice being input by the second user, providing a secondadditional response sound obtained by inputting the second additionaluser's voice and second utterance history information corresponding tothe second user's voice to the artificial intelligence model to bedistinct from the first additional response sound.

The providing a first additional response sound may include a first UIcorresponding to the first user while providing the first additionalresponse sound, and the providing a second additional response sound tobe distinct from the first additional response sound may includedisplaying a second UI corresponding to the second user while providingthe second additional response sound.

The method may further include based on the user who uttered the firstuser's voice being different from the user who uttered the second user'svoice, obtaining a first domain corresponding to the first user's voiceand a second domain corresponding to the second user's voice, and basedon the first domain being the same as the second domain, providing thesecond response sound obtained by inputting the second user's voice andthe utterance history information to the artificial intelligence model.

In the method, based on the second user's voice being input within afirst predetermined period of time from a point when the first user'svoice is input, the user who uttered the first user's voice being thesame as the user who uttered the second user's voice, and a first domaincorresponding to the first user's voice being different from a seconddomain corresponding to the second user's voice, the first user's voicemay be stored in the storage without providing the first response sound,and a third response sound obtained by inputting the second user's voiceto the artificial intelligence model may be provided.

The method may further include based on a user's voice corresponding tothe second domain not being input from the user for a secondpredetermined period of time or longer, providing the first responsesound obtained by inputting the first user's voice to the artificialintelligence model.

The identifying a user may include identifying the user based on atleast one of a captured image or an input user's voice.

The identifying a user who uttered the first user's voice may includeidentifying the user who uttered the first user's voice based on a shapeof mouth of a user included in the image for a period of time duringwhich the first user's voice is input, and the providing a secondresponse sound may include, based on the shape of mouth of theidentified user included in the image being changed for a period of timeduring which the second user's voice is input, identifying the user whouttered the second user's voice to be the same as the user who utteredthe first user's voice.

Effect of Invention

According to the embodiments of the disclosure described above, whenuser's voices are input from a plurality of users, respectively, theelectronic device is able to provide a response while maintaining acontext by processing, postponing, or separately processing a pluralityof user's voices based on the existing user's voices.

DESCRIPTION OF DRAWINGS

FIGS. 1A and 1B are views for explaining a method for processing aplurality of user's voices according to an embodiment.

FIG. 2 is a view illustrating an electronic system which provides aresponse sound to a user's voice according to an embodiment.

FIG. 3A is a block diagram for explaining a configuration of anelectronic device according to an embodiment.

FIG. 3B is a block diagram illustrating a specific configuration of theelectronic device according to an embodiment.

FIGS. 4A and 4B are views for explaining postponement of processing of avoice of a user among a plurality of users according to an embodiment.

FIGS. 5A to 5C are views for explaining a method for real-timesimultaneous processing of a plurality of user's voices according to anembodiment.

FIG. 6 is a view for explaining a collective interaction with aplurality of users according to an embodiment.

FIG. 7 is a view for explaining an operation when a plurality of user'svoices respectively corresponding to a plurality of domains of the sameuser are input, according to an embodiment.

FIGS. 8A and 8B are views for explaining a method for identifying a userwho uttered a user's voice according to an embodiment.

FIGS. 9A and 9B are views for explaining additional training of anartificial intelligence model according to an embodiment.

FIG. 10 is a flowchart for explaining operations of an electronic devicestoring an artificial intelligence model according to an embodiment.

FIG. 11 is a sequence diagram for explaining operations of theelectronic device and an external server storing an artificialintelligence model according to an embodiment.

FIG. 12 is a block diagram illustrating a configuration of an externalserver for training and using an artificial intelligence model whichprovides a response to a user's request according to an embodiment.

FIG. 13 is a block diagram illustrating a learning unit according to anembodiment.

FIG. 14 is a block diagram illustrating a response unit according to anembodiment.

FIG. 15 is a view illustrating an example in which an electronic deviceand an external server are linked to each other to train and identifydata according to an embodiment.

FIG. 16 is a flowchart for explaining a method for controlling anelectronic device according to an embodiment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, various embodiments of the disclosure will be describedwith reference to the accompanying drawings. It should be noted that thetechnologies disclosed in this disclosure are not for limiting the scopeof the disclosure to a specific embodiment, but they should beinterpreted to include all modifications, equivalents and/oralternatives of the embodiments of the disclosure. In relation toexplanation of the drawings, similar reference numerals may be used forsimilar elements.

In this disclosure, the terms such as “comprise”, “may comprise”,“consist of”, or “may consist of” are used herein to designate apresence of corresponding features (e.g., constituent elements such asnumber, function, operation, or part), and not to preclude a presence ofadditional features.

In this disclosure, expressions such as “A or B”, “at least one of A[and/or] B,”, or “one or more of A [and/or] B,” include all possiblecombinations of the listed items. For example, “A or B”, “at least oneof A and B,”, or “at least one of A or B” includes any of (1) at leastone A, (2) at least one B, or (3) at least one A and at least one B.

The expressions “first,” “second” and the like used in the disclosuremay denote various elements, regardless of order and/or importance, andmay be used to distinguish one element from another, and does not limitthe elements.

If it is described that a certain element (e.g., first element) is“operatively or communicatively coupled with/to” or is “connected to”another element (e.g., second element), it should be understood that thecertain element may be connected to the other element directly orthrough still another element (e.g., third element). On the other hand,if it is described that a certain element (e.g., first element) is“directly coupled to” or “directly connected to” another element (e.g.,second element), it may be understood that there is no element (e.g.,third element) between the certain element and the another element.

Also, the expression “configured to” used in the disclosure may beinterchangeably used with other expressions such as “suitable for,”“having the capacity to,” “designed to,” “adapted to,” “made to,” and“capable of,” depending on cases. Meanwhile, the expression “configuredto” does not necessarily mean that a device is “specifically designedto” in terms of hardware. Instead, under some circumstances, theexpression “a device configured to” may mean that the device “is capableof” performing an operation together with another device or component.For example, the phrase “a unit or a processor configured (or set) toperform A, B, and C” may mean a dedicated processor (e.g., an embeddedprocessor) for performing the corresponding operations, or ageneric-purpose processor (e.g., a CPU or an application processor) thatcan perform the corresponding operations by executing one or moresoftware programs stored in a memory device.

An electronic device according to various embodiments of the disclosuremay include at least one of, for example, a smartphone, a tablet PC, amobile phone, a video phone, an e-book reader, a desktop PC, a laptopPC, a netbook computer, a workstation, a server, a PDA, a portablemultimedia player (PMP), an MP3 player, a mobile medical device, acamera, or a wearable device. A wearable device may include at least oneof an accessory type (e.g., a watch, a ring, a bracelet, an anklebracelet, a necklace, a pair of glasses, a contact lens or ahead-mounted-device (HMD)); a fabric or a garment-embedded type (e.g.:electronic cloth); skin-attached type (e.g., a skin pad or a tattoo); ora bio-implant type (implantable circuit). In addition, in someembodiments, the electronic device may include at least one of, forexample, a television, a digital video disk (DVD) player, an audiosystem, a refrigerator, air-conditioner, a vacuum cleaner, an oven, amicrowave, a washing machine, an air purifier, a set top box, a homeautomation control panel, a security control panel, a media box (e.g.,SAMSUNG HOMESYNC™, APPLE TV™, or GOOGLE TV™), a game console (e.g.,XBOX™, PLAYSTATION™), an electronic dictionary, an electronic key, acamcorder, or an electronic frame.

In other embodiments, the electronic device may include at least one ofa variety of medical devices (e.g., various portable medical measurementdevices such as a blood glucose meter, a heart rate meter, a bloodpressure meter, or a temperature measuring device, magnetic resonanceangiography (MRA), magnetic resonance imaging (MRI), or computedtomography (CT) scanner, or ultrasonic wave device, etc.), a navigationsystem, a global navigation satellite system (GNSS), an event datarecorder (EDR), a flight data recorder (FDR), an automotive infotainmentdevice, a marine electronic equipment (e.g., marine navigation devices,gyro compasses, etc.), avionics, a security device, a car head unit,industrial or domestic robots, a drone, an ATM of financial institution,a point of sale of (POS) a store, or an Internet of Things (IoT) device(e.g., light bulbs, sensors, sprinkler devices, fire alarms,thermostats, street lights, toasters, exercise equipment, hot watertanks, heater, boiler, etc.).

In this disclosure, a term “user” may refer to a person using anelectronic device or a device (e.g., an artificial intelligenceelectronic device) using an electronic device.

FIGS. 1A and 1B are views for explaining a method for processing aplurality of user's voices according to an embodiment of the disclosure.

As illustrated in FIG. 1A, an electronic device may receive a user'svoice uttered by a user A. For example, the electronic device mayreceive a user's voice of “Add college reunion in Gangnam in calendar”uttered by the user A.

The electronic device may transmit the user's voice of the user A to anexternal server. The external server may include a first server forconverting the user's voice into a text and a second server forproviding response information to the user's voice based on theconverted text, but this is merely an embodiment, and the externalserver may be implemented as a single server.

The external server may convert the user's voice received from theelectronic device into a text through the Speech-to-Text (STT)technology. The external server may obtain response information to theuser's voice by inputting the converted text to an artificialintelligence model trained through an artificial intelligence algorithm.The response information may be a response sound including a response tothe user's voice and may be transmitted to the electronic device asaudio data. Alternatively, the response information may be textinformation obtained by the second server and may be converted into aresponse sound by the first server by using the Text-to-Speech (TTS)technology and transmitted to the electronic device. However, there isno limitation thereto, and the response information may further includeinformation regarding an image, in addition to the information regardingthe voice. In such a case, the information regarding the image may beinformation such as a text, an image, an URL, or the like including aresponse to the user's voice. In addition, the information regarding thevoice may be the same as the information regarding the image, but thisis merely an embodiment, and the information regarding the voice mayinclude only a part of the information regarding the image or may besummarized information of the information regarding the image.

The external server may transmit the response sound to the user's voiceto the electronic device. For example, the external server may transmita response sound of “What date and what time is it?” as a response tothe user's voice of the user A to the electronic device. The electronicdevice may provide the received response sound.

However, the function of providing the response sound by the externalserver may be executed by the electronic device. For example, theelectronic device may convert a user's voice uttered by the user A intoa text, obtain response information to the user's voice by inputting theconverted text to an artificial intelligence model, change the responseinformation to a response sound, and provide the response sound to theuser A.

Hereinafter, the electronic device may receive a user's voice uttered bya user B. For example, the electronic device may receive a user's voiceof “How is the weather today?” uttered by a user B.

The electronic device may determine whether or not the users are thesame by comparing the user's voice of the user A and the user's voice ofthe user B. For example, the electronic device may determine whether ornot the users are the same based on a voiceprint of each of the user'svoice of the user A and the user's voice of the user B. If the users aredifferent from each other, the electronic device may store the user'svoice of the user B in a storage and may not transmit the user's voiceto the external server.

Alternatively, the external server may determine whether or not theusers are the same. For example, the electronic device may transmit theuser's voice of the user B to the external server, without determiningwhether or not the users are the same. The external server may determinewhether or not the users are the same by comparing the received user'svoice of the user B and the previous user's voice of the user A. If theusers are different from each other, the external server may store theuser's voice of the user B in a storage. In such a case, the externalserver may not generate a response sound to the user's voice of the userB. The electronic device may not receive any information from theexternal server and may not provide a response sound to the user's voiceof the user B. In addition, the external server may be in a situation ofproviding the response to the user A and may transmit a response soundguiding the user B to wait to the electronic device. The electronicdevice may provide the received response sound to the user B.

Then, the electronic device may receive a user's voice uttered by theuser A. For example, the electronic device may receive a user's voice of“3 o'clock on 17th of this month” uttered by the user A.

The electronic device may provide a response sound to the user's voiceof the user A by the same method as described above. For example, theelectronic device may provide a response sound of “Added at 3 o'clock on17th of December” to the user's voice of the user A.

Then, as illustrated in FIG. 1B, if a user's voice is not input from theuser A for a predetermined period of time, the electronic device maytransmit the user's voice of “How is the weather today?” uttered by theuser B to the external server. The external server may transmit aresponse sound of “Today's weather you asked a while ago is fine” to theuser's voice of the user B to the electronic device by the same methodas described above. The electronic device may provide the receivedresponse sound to the user B. The electronic device may identify thepresence of the user B and provide the response sound. If the presenceof the user B is not identified, the electronic device may store theresponse sound in the storage and may provide the response sound, if thepresence of the user B is identified later.

In addition, when the external server stores the user's voice of theuser B and the external server provides the response sound of “Added at3 o'clock on 17th of December” to the electronic device, the externalserver may determine that the interaction with the user A has ended.When it is determined that the interaction with the user A has ended,the external server may transmit the response sound of “Today's weatheryou asked a while ago is fine” to the user's voice of the user B storedin the storage to the electronic device.

The trained model provided by the external server as described above maybe, for example, a model based on a neural network as a determinationmodel trained based on an artificial intelligence algorithm. The traineddetermination model may be designed to simulate a brain structure ofhuman on the computer and may include a plurality of network nodesincluding weights and simulating neurons of the neural network of human.The plurality of network nodes may form connections to each other tosimulate a synaptic activity in that the neurons transmit and receivesignals through synapse. In addition, the trained determination modelmay include, for example, a neural network model or a deep learningmodel developed from the neural network model. In a deep learning model,a plurality of network nodes may be positioned at depths (or on layers)different from each other and may transmit and receive data according tothe convolution connection. Examples of the trained determination modelmay include a deep neural network (DNN), a recurrent neural network(RNN), and a bidirectional recurrent deep neural network (BRDNN), butthere is no limitation thereto.

In addition, the electronic device may use a personal assistant programwhich is an artificial intelligence dedicated program (or artificialintelligence agent) in order to receive the responses to the user'svoices as sounds and images as described above. The personal assistantprogram may be executed by the existing generic-purpose processor (e.g.,CPU) or a separate AI dedicated processor (e.g., GPU) as a dedicatedprocessor for providing artificial intelligence (AI)-based service.

Specifically, the artificial intelligence agent may be operated (orexecuted) when a predetermined user input (e.g., user utterancecorresponding to a text indicating an electronic device or the like) isinput or a button (e.g., button for executing the artificialintelligence agent) provided on the electronic device is pressed. Theartificial intelligence agent may transmit an input user's voice to anexternal server, provide a response sound to the user's voice via aspeaker based on information regarding a voice received from theexternal server, and provide a guide screen including the response tothe user's voice based on the information regarding an image receivedfrom the external server.

When a predetermined user input is detected on a screen or a button(e.g., button for executing the artificial intelligence agent) providedon the electronic device is pressed, the artificial intelligence agentmay be operated. In addition, the artificial intelligence agent may bein a state of being already executed before the predetermined user inputis detected or the button provided on the electronic device is pressed.In this case, the artificial intelligence agent of the electronic devicemay provide a voice and an image including the response to the user'svoice after the predetermined user input is detected or the buttonprovided on the electronic device is pressed. In addition, theartificial intelligence agent may be in a standby state before thepredetermined user input is detected or the button provided on theelectronic device is pressed. The standby state is a state of detectingreception of a predefined user input in order to control the start ofthe operation of the artificial intelligence agent. When thepredetermined user input is detected or the button provided on theelectronic device is pressed while the artificial intelligence agent isin the standby state, the electronic device may operate the artificialintelligence agent and provide a voice and an image including theresponse to the user's voice.

According to another embodiment of the disclosure, when the electronicdevice directly obtains information regarding a voice and an imageincluding a response to a user's voice by using an artificialintelligence model, the artificial intelligence agent may obtaininformation regarding a voice and an image by controlling an artificialintelligence model. The artificial intelligence agent may execute theoperation of the external server described above.

FIG. 2 is a view illustrating an electronic system which provides aresponse sound to a user's voice according to an embodiment of thedisclosure. As illustrated in FIG. 2, the electronic system may includean electronic device 100 and an external server 200.

The electronic device 100 may store an artificial intelligence dedicatedprogram. The artificial intelligence dedicated program may beimplemented in various aspects.

First, regarding an artificial intelligence dedicated program accordingto a first aspect, the electronic device 100 may be used in order toreceive a user's voice and provide a response sound to the user's voice.In such a case, the electronic device 100 may transmit an input user'svoice to the external server 200, receive a response sound to the user'svoice from the external server 200, and provide the received responsesound to the user.

The external server 200 may identify a user from the user's voicereceived from the electronic device 100. For example, the externalserver 200 may identify a user from a voiceprint of the user's voice.However, there is no limitation thereto, and any method for identifyinga user from the user's voice may be used.

The external server 200 may determine a domain of the user's voice. Forexample, when a user's voice of “How is the weather today?” is received,the external server 200 may identify that a domain of the receiveduser's voice is an information request domain. In addition, when auser's voice of “Turn the latest music on” is received, the external 200may identify that a domain of the received user's voice is a devicecontrol domain.

Further, the external server 200 may determine whether or not a contextof the received user's voice coincides with a context of utterancehistory information. For example, if a user's voice of “How is theweather today?” is received and then a user's voice of “What abouttomorrow?” is received, the external server 200 may determine that thecontexts coincide with each other. In addition, if a user's voice of“How is the weather today?” is received and then a user's voice of“What's on TV now?” is received, the external server 200 may determinethat the contexts does not coincide with each other.

The external server 200 may generate response information based on theidentified user, the domain of the received voice, and whether or notthe contexts thereof coincide with each other. That is, the informationindicating the identified user, the domain of the received voice, andwhether or not the contexts thereof coincide with each other may beinput to an artificial intelligence model stored in the external server200 together with the user's voice.

However, this is merely an embodiment and only the user's voice may beinput to the artificial intelligence model. For example, the artificialintelligence model may include a first artificial intelligence sub-modelwhich identifies a user from an input user's voice and a secondartificial intelligence sub-model which generates response informationfrom the user's voice.

In such a case, the second artificial intelligence sub-model may notdetermine the domain of the user's voice and whether or not the contextscoincide with each other. However, a sample dialogue pattern or the likeused in a training process of the second artificial intelligencesub-model may be generally a dialogue pattern with the coincidingdomains and contexts. Accordingly, when a user's voice having a domainor a context different from that of the utterance history information isinput, the second artificial intelligence sub-model may separatelyprocess the input user's voice or postpone processing of the user'svoice.

That is, in a case of using the artificial intelligence model, theexternal server 200 may not specifically determine the domain and thecontext of the user's voice. However, the external server 200 maygenerate response information, to which the domain and the context ofthe user's voice are substantially reflected, due to the effect of thesample dialogue pattern or the like used in the training process.

The external server 200 may change the generated response information toa response sound and transmit the response sound to the electronicdevice 100.

Regarding an artificial intelligence dedicated program according to asecond aspect, the electronic device 100 may be used in order to receivea user's voice, identify a user, and provide a response sound to theuser's voice. In such a case, the electronic device 100 may identify auser who uttered an input user's voice and transmit the input user'svoice and the identified user information to the external server 200.The electronic device 100 may receive a response sound to the user'svoice from the external server 200 and provide the received responsesound to the user.

The external server 200 may generate response information by inputtingthe user's voice and the user information to an artificial intelligencealgorithm. The external server 200 may manage utterance historyinformation based on the user information. For example, when the user'svoice of the user A are received, the external server 200 may generateresponse information by inputting the utterance history information ofthe user A and the user's voice of the user A to the artificialintelligence algorithm, and when the user's voice of the user B arereceived, the external server 200 may generate response information byinputting the utterance history information of the user B and the user'svoice of the user B to the artificial intelligence algorithm.

The external server may change the generated response information to aresponse sound and transmit the response sound to the electronic device100.

An artificial intelligence dedicated program according to a third aspectmay be a program including an artificial intelligence model. In such acase, the electronic device 100 may identify a user who uttered an inputuser's voice and generate response information by inputting the inputuser's voice and the identified user information to the artificialintelligence model. The electronic device 100 may change the responseinformation to a response sound and output the response sound via aspeaker. That is, the electronic device 100 may provide the responsesound to the user's voice without the external server 200.

FIG. 3A is a block diagram for explaining a configuration of anelectronic device 100 according to an embodiment of the disclosure.

As illustrated in FIG. 3A, the electronic device 100 may include amicrophone 110, a speaker 120, and a processor 130.

The electronic device 100 may be a device which receives a user's voiceand provides various interactions according to the received user'svoice. For example, the electronic device 100 may be a speaker, anotebook, a PC, a tablet PC, or a smartphone. However, there is nolimitation thereto, and the electronic device 100 may be any device, aslong as it is able to receive a user's voice and provide a responsesound according thereto.

In addition, the electronic device 100 may be a device including adisplay instead of the speaker 120. In such a case, the electronicdevice 100 may receive a user's voice and provide an interactionaccording to the received user's voice via the display. In addition, theelectronic device 100 may be any device, as long as it is able toreceive a user's voice and provide response information to the receiveduser's voice.

The microphone 110 is a component for receiving a user's voice or othersounds and converting it into audio data. The microphone 110 may beelectrically connected to the processor 130 and receive a user's voiceunder the control of the processor 130.

The microphone 110 may include a plurality of sub-microphones. Forexample, the microphone 110 may include one sub-microphone at each offront, back, right, and left portions of the electronic device 100.However, there is no limitation thereto, and the electronic device 100may include only one microphone 110.

The speaker 120 is a component which outputs not only various pieces ofaudio data but also various alerts or voice messages. The speaker 120may be electrically connected to the processor 130 and output a responsesound under the control of the processor 130.

The speaker 120 may include a plurality of sub-speakers. For example,the speaker 120 may include one sub-speaker at each of right and leftportions of the electronic device 100. However, there is no limitationthereto and the electronic device 100 may include only one speaker 120.

The processor 130 may generally control the operations of the electronicdevice 100.

According to an embodiment, the processor 130 may be implemented as adigital signal processor (DSP), a microprocessor, or a time controller(TCON). However, there is no limitation thereto, and the processor 130may include one or more of a central processing unit (CPU), amicrocontroller unit (MCU), a microprocessing unit (MPU), a controller,an application processor (AP), or a communication processor (CP), and anARM processor or may be defined as the corresponding term. In addition,the processor 130 may be implemented as System on Chip (SoC) or largescale integration (LSI) including the processing algorithm or may beimplemented in form of a Field Programmable gate array (FPGA).

When a first user's voice is input via the microphone 110, the processor130 may identify a user who uttered the first user's voice and provide afirst response sound obtained by inputting the first user's voice to anartificial intelligence model trained through an artificial intelligencealgorithm via the speaker 120.

The artificial intelligence model may be stored in the electronic device100, and in such a case, the processor 130 may obtain the first responsesound by inputting the first user's voice to the artificial intelligencemodel.

In addition, the artificial intelligence model may be stored in theexternal server 200, and in such a case, the processor 130 may transmitthe first user's voice to the external server 200 and receive the firstresponse sound obtained by inputting the first user's voice to theartificial intelligence model from the external server 200.

When a second user's voice is input via the microphone 110, theprocessor 130 may identify a user who uttered the second user's voice,and if the user who uttered the first user's voice is the same as theuser who uttered the second user's voice, the processor 130 may providea second response sound obtained by inputting the second user's voiceand the utterance history information to the artificial intelligencemodel via the speaker 120.

For example, the processor 130 may receive the second user's voiceuttered by the user who uttered the first user's voice, and in such acase, the processor 130 may provide the second response sound obtainedby inputting not only the second user's voice but also the utterancehistory information to the artificial intelligence model. That is, thesecond response sound may be obtained by reflecting not only thecurrently input user's voice, but also all of the previously inputuser's voice and response information according to user's voice.

The electronic device 100 may further include a camera, and theprocessor 130 may identify a user based on at least one of an imagecaptured by the camera or a user's voice input via the microphone.

Specifically, the processor 130 may identify the user who uttered thefirst user's voice by identifying a face of the user included in theimage for a period of time during which the first user's voice is input.

In addition, the processor 130 may identify the user who uttered thefirst user's voice based on a shape of mouth of a user included in theimage for a period of time during which the first user's voice is input.

For example, the processor 130 may capture an image by the camera inreal time, and identify a user who changed the shape of mouth among aplurality of users included in an image which is being captured in realtime for a period of time during which the first user's voice is input,as the user who uttered the first user's voice.

If the shape of mouth of the identified user included in the image ischanged for a period of time during which the second user's voice isinput is changed, the processor 130 may identify that the user whouttered the first user's voice is the same as the user who uttered thesecond user's voice.

In addition, the processor 130 may identify whether or not the users arethe same by using the speaker 120. For example, the speaker 120 mayinclude a plurality of sub-speakers, and the processor 130 may identifya position of a user who uttered a first user's voice by using theplurality of sub-speakers, when the first user's voice is input.

When a second user's voice is input, the processor 130 may identify aposition of a user who uttered the second user's voice by using theplurality of sub-speakers and determine whether or not the user haschanged by comparing the position of the user identified previously.

In addition, the processor 130 may identify the user through avoiceprint of a user's voice. For example, the electronic device 100 mayfurther include a storage electrically connected to the processor 130and the processor 130 may execute frequency analysis of each of aplurality of user's voices and store frequency characteristics of eachof the plurality of user voices in the storage. Then, when the firstuser's voice is input, the processor 130 may execute frequency analysisof the first user's voice and identify a user who uttered the firstuser's voice based on information stored in the storage. The processor130 may execute the same operation with respect to the second user'svoice and determine whether or not the user is the same as theidentified user.

The operation of using the voiceprint may also be executed by theexternal server 200. In such a case, the electronic device 100 may notexecute the operation of identifying the user and may transmit the inputuser's voice to the external server 200. The external server 200 mayidentify a user who uttered the received user's voice by executing thefrequency analysis of the received user's voice.

In addition, the processor 130 may not identify the user who uttered theinput user's voice but may classify gender, kid/adult/elderly person,and identify regions, and the like. For example, when the first user'svoice is input, the processor 130 may identify that a user who utteredthe first user's voice is an adult man and he speaks the standardlanguage. When the second user's voice is input, the processor 130 mayidentify that a user who uttered the second user's voice is an adultwoman and she speaks the standard language. In this case, the processor130 may identify that the users are not the same.

The classification of gender, kid/adult/elderly person, and theidentification of regions, and the like may also be executed by theexternal server 200. In this case, the electronic device 100 may notexecute the operation of identifying the user and may transmit the inputuser's voices to the external server 200. The external server 200 mayclassify gender, kid/adult/elderly person, and identify regions, and thelike by using the received user's voice and may determine whether or notthe users are the same.

In the above, the processor 130 may store the input user's voice and theresponse sound to the user's voice in the storage as the utterancehistory information for a predetermined period of time from a point whenthe user's voice is input. For example, when a user's voice of “How isthe weather going to be tomorrow?” is input, the processor 130 mayprovide a response sound of “Tomorrow's weather will be fine” to theuser's voice and store interaction information as the utterance historyinformation in the storage for 10 minutes from the point when the user'svoice is input. When an additional user's voice is input from the sameuser within 10 minutes from the point when the user's voice is input,the processor 130 may provide a response sound obtained by inputting theadditional user's voice and the utterance history information to theartificial intelligence model. In addition, when the additional user'svoice is input from the same user after 10 minutes from the point whenthe user's voice is input, the processor 130 may provide a responsesound obtained by inputting only the additional user's voice to theartificial intelligence model, without determining whether or not theuser who uttered the additional user's voice is the same as the user whouttered the initial user's voice. That is, the processor 130 maydetermine whether or not the user who uttered the additional user'svoice is the same as the user who uttered the initial user's voice, onlywhen the additional user's voice is input within a predetermined periodof time from the point when the user's voice is input.

When the additional user's voice is input within the predeterminedperiod of time from the point when the user's voice is input, theprocessor 130 may change the start point of the predetermined period oftime as the point when the additional user's voice is input. Forexample, when the first user's voice is input at 6 p.m. and the seconduser's voice is input at 6:05 p.m. which is within the predeterminedperiod of time of 10 minutes from 6 p.m., the processor 130 may storethe utterance history information from 6:05 p.m. to 6:15 p.m. which isafter the predetermined period of time which is 10 minutes. Theutterance history information may include the first user's voice, thefirst response sound to the first user's voice, the second user's voice,and the second response sound to the second user's voice. That is, whena period of time for storing the utterance history information isextended by the continuous utterance of the user's voice by the user,the processor 130 may maintain the utterance history information to theextended time to be stored in the storage.

However, there is no limitation thereto, and although the period of timefor storing the utterance history information is extended by thecontinuous utterance of the user's voice by the user, the utterancehistory information until the extended time may not be maintained to bestored in the storage. For example, when the period of time for storingthe utterance history information is extended by the continuousutterance of the user's voice by the user, the processor 130 may storeonly the user's voice and the response sound from a past point before apredetermined period of time from a current point to the current pointas the utterance history information. That is, the processor 130 mayremove the user's voice which has passed the predetermined period oftime from an utterance point and the response sound which has passed thepredetermined period of time from a sound providing point from theutterance history information from the storage.

When the electronic device 100 stores the artificial intelligence model,the storage may store the utterance history information for each user ina time sequence. For example, the storage may store utterance historyinformation on Jan. 1, 2017, utterance history information on Jan. 5,2017 of the user A, utterance history information the user B on Jan. 4,2017, and the like. The processor 130 may transmit the utterance historyinformation which has passed the predetermined period of time from theutterance history information stored in the storage to the externalserver 200. The processor 130 may receive the utterance historyinformation transmitted to the external server 200 from the externalserver 200, if necessary. The processor 130 may extract a keyword or thelike of the utterance history information and store the extractedkeyword in the storage before transmitting the utterance historyinformation to the external server 200, and may transmit the utterancehistory information to the external server 200. The extracted keywordmay be used when a user searches for past utterance history informationlater.

For example, when a user utters a user's voice of “Which restaurant wasit that we searched to go in Jeju Island about 5 years ago?”, theprocessor 130 may search for Jeju Island and restaurant as keywords fromthe keywords of 5 years ago and receive corresponding utterance historyinformation from the external server 200. The processor 130 may provideinformation desired by a user from the utterance history informationreceived from the external server.

The electronic device 100 may further include a storage. When the userwho uttered the first user's voice is not the same as the user whouttered the second user's voice, the processor 130 may store the seconduser's voice in the storage, and when a user's voice is not input fromthe user who uttered the first user's voice, for a predetermined periodof time or longer, the processor 130 may provide a third response soundobtained by inputting the second user's voice to the artificialintelligence model via the speaker 120.

For example, when the user B utters the second user's voice within thepredetermined period of time from the point when the user A uttered thefirst user's voice, the processor 130 may determine whether or not theusers are the same, and when it is determined that the users are not thesame, the processor 130 may store the second user's voice of the user Bin the storage. That is, the processor 130 may postpone providing theresponse sound to the second user's voice of the user B. When theadditional user's voice of the user A is not input until thepredetermined period of time passes from the point when the user A hasuttered the first user's voice, the processor 130 may provide theresponse sound to the second user's voice after the predetermined periodof time from the point when the user has uttered the first user's voice.The response sound to the second user's voice may be a response soundobtained by inputting the second user's voice to the artificialintelligence model in a state where the utterance history information bythe user A is removed.

When the first user's voice is input, the processor 130 may identify thefirst user who uttered the first user's voice and provide the firstresponse sound obtained by inputting the first user's voice to theartificial intelligence model via the speaker 120, and when the seconduser's voice is input, the processor 130 may identify the second userwho uttered the second user's voice and provide the third response soundobtained by inputting the second user's voice to the artificialintelligence model to be distinct from the first response sound via thespeaker 120. That is, the processor 130 may separately process theplurality of user's voices in real time and set providing states ofprocessed results to be distinct from each other.

For example, the processor 130 may provide “Today's weather is fine” asthe first response sound to “How is the weather today?” of the user A ina male voice and provide “No movie tonight” as the second response soundto “What movie is on tonight?” of the user B in a female voice. Thesecond response sound may be a response sound obtained only based on theuser's voice of the user B without reflecting the utterance historyinformation of the user A.

However, there is no limitation thereto, and the processor 130 mayprovide the providing states of the response information to the user'svoices to be distinct from each other by any methods. For example, theelectronic device 100 may further include a display electricallyconnected to the processor 130, and the processor 130 may display“Today's weather is fine” as the first response to “How is the weathertoday?” of the user A via the display and provide “No movie tonight” asthe second response sound to “What movie is on tonight?” of the user Bvia the speaker 120.

In addition, the processor 130 may display both of “Today's weather isfine” as the first response to “How is the weather today?” of the user Aand “No movie tonight” as the second response to “What movie is ontonight?” of the user B, and may display different background screencolors of the first response and the second response so that the usersmay identify the target of the response.

When a first additional user's voice is input by the first user, theprocessor 130 may provide a first additional response sound obtained byinputting the first additional user's voice and first utterance historyinformation corresponding to the first user's voice to the artificialintelligence model via the speaker 120, and when a second additionaluser's voice is input by the second user, the processor 130 may providea second additional response sound obtained by inputting the secondadditional user's voice and second utterance history informationcorresponding to the second user's voice to the artificial intelligencemodel via the speaker 120 to be distinct from the first additionalresponse sound.

The processor 130 may display a first UI corresponding to the first uservia the display while providing the first additional response sound, anddisplay a second UI corresponding to the second user via the displaywhile providing the second additional response sound.

For example, the processor 130 may display an image of the first uservia the display while providing the first additional response sound andmay display an image of the second user via the display while providingthe second additional response sound.

In a case of providing the response sound for each user by using theexternal server 200 storing an artificial intelligence algorithm, theprocessor 130 may transmit the user's voices to the external server 200together with the identified user information. The external server 200may generate utterance history information for each user based on thereceived user information and transmit a response sound obtained byinputting the user's voice and the corresponding utterance historyinformation to the artificial intelligence model to the electronicdevice 100. The external server 200 may transmit not only the responsesound but also the user information of a target of the response sound tothe electronic device 100.

The electronic device 100 may provide the response sound based on thereceived user information to be distinct from the response sound of theother user.

When the user who uttered the first user's voice is not the same as theuser who uttered the second user's voice, the processor 130 may obtain afirst domain corresponding to the first user's voice and a second domaincorresponding to the second user's voice, and when the first domain isthe same as the second domain, the processor 130 may provide the secondresponse sound obtained by inputting the second user's voice and theutterance history information to the artificial intelligence model viathe speaker 120.

For example, the processor 130 may provide “It will rain todayafternoon” as the first response sound to the first user's voice of “Howis the weather today?” of the user A via the speaker 120, and when thesecond user's voice of “Do I need to take umbrella?” of the user B isinput, the processor 130 may determine whether or not the users aresame. When it is determined that the users are not same, the processor130 may obtain that both of a domain of the first user's voice and adomain of the second user's voice are information request domains. Whenit is determined that the domain of the first user's voice is the sameas the domain of the second user's voice, the processor 130 may providethe second response sound of “Yes. Take umbrella today” obtained byinputting the second user's voice and the utterance history informationto the artificial intelligence model.

In the example described above, when the second user's voice of “Turnthe music on” is uttered by the user B, the processor 130 may determinewhether or not the users are same. When it is determined that the usersare not same, the processor 130 may obtain that the domain of the firstuser's voice is the information request domain and obtain that thedomain of the second user's voice is a device control domain. When it isdetermined that the domain of the first user's voice is the differentfrom the domain of the second user's voice, the processor 130 maypostpone the processing of the second user's voice. In addition, theprocessor 130 may provide the second response sound of “Turning musicon” obtained by inputting only the second user's voice to the artificialintelligence model to be distinct from the first response sound.

In addition, the processor 130 may further consider the context, inaddition to the domain. For example, although the domain of the firstuser's voice is same as the second user's voice, if the contexts thereofare different from each other, the processor 130 may postpone theprocessing of the second user's voice. In addition, the processor 130may provide the second response sound obtained by inputting only thesecond user's voice to the artificial intelligence model to be distinctfrom the first response sound.

When the second user's voice is input within a first predeterminedperiod of time from the point when the first user's voice is input, theuser who uttered the first user's voice is the same as the user whouttered the second user's voice, and the first domain corresponding tothe first user's voice is different from the second domain correspondingto the second user's voice, the processor 130 may store the first user'svoice in the storage without providing the first response sound, andprovide the third response sound obtained by inputting the second user'svoice to the artificial intelligence model via the speaker. The firstpredetermined period of time may be shorter than the period of timeduring which the storage of the utterance history information describedabove is maintained.

For example, when the first user's voice of “Turn TV on” of the user Ais input and then the second user's voice of “Where is the place of theappointment today?” with the domain different from that of the firstuser's voice of the same user A is input within 1 second, the processor130 may store the first user's voice in the storage and provide “Infront of City Hall” as the third response sound to the second user'svoice.

When the user's voice corresponding to the second domain is not inputfrom the user for a second predetermined period of time or longer, theprocessor 130 may provide the first response sound obtained by inputtingthe first user's voice stored in the storage to the artificialintelligence model via the speaker 120. The second predetermined periodof time may be same as the period of time for maintaining the storage ofthe utterance history information described above and longer than thefirst predetermined period of time. However, there is no limitationthereto, and the second predetermined period of time may not correlateto the period of time for maintaining the storage of the utterancehistory information described above and may be longer than the firstpredetermined period of time. In addition, in the above example, theprocessor 130 may determine whether or not the context of the seconduser's voice is the same as the context of the first user's voice, andwhen the contexts are different, the processor 130 may store the firstuser's voice in the storage without providing the first response sound,and may provide the third response sound obtained by inputting thesecond user's voice to the artificial intelligence model via thespeaker.

The artificial intelligence algorithm stored in the electronic device100 or the external server 200 may include at least one of a firstartificial intelligence algorithm for obtaining a domain of a user'svoice or a second artificial intelligence algorithm for determiningwhether or not the contexts are the same between the user's voices. Inthis case, the determination whether or not the domains and the contextsare the same may be performed before obtaining the response information.

In addition, the artificial intelligence algorithm stored in theelectronic device 100 or the external server 200 may not include theseparate artificial intelligence algorithm for obtaining the domain ofthe user's voice or determining whether or not the contexts are the samebetween the user's voices. In this case, the artificial intelligencealgorithm may provide a response sound obtained by reflecting at leastone of the domain or the context as it is.

FIG. 3B is a block diagram illustrating a specific configuration of theelectronic device 100 according to an embodiment of the disclosure. Theelectronic device 100 may include the microphone 110, the speaker 120,and the processor 130. In addition, referring to FIG. 3B, the electronicdevice 100 may further include a communicator 140, a storage 145, adisplay 150, a user interface 155, a camera 160, an audio processor 170,a video processor 180, and a button 190. The specific description of thecomponents illustrated in FIG. 3B which are overlapped with thecomponents illustrated in FIG. 3A will be omitted.

The processor 130 generally controls the operations of the electronicdevice 100 by using various programs stored in the storage 145.

Specifically, the processor 130 includes a RAM 131, a ROM 132, a mainCPU 133, a graphic processor 134, first to n-th interfaces 135-1 to135-n, and a bus 136.

The RAM 131, the ROM 132, the main CPU 133, the graphic processor 134,and the first to n-th interfaces 135-1 to 135-n may be connected to eachother via the bus 136.

The first to n-th interfaces 135-1 to 135-n are connected to the variouscomponents described above. One of the interfaces may be a networkinterface connected to an external device via the network.

The main CPU 133 may execute the booting by using the O/S stored in thememory 145 by accessing the storage 145. The main CPU 133 may executevarious operations by using various programs and the like stored in thestorage 145.

The ROM 132 may store a set of instructions for system booting. If aturn-on instruction is input to supply power, the main CPU 133 copiesthe O/S stored in the storage 145 to the RAM 131 and boots the system upby executing the O/S according to the instruction stored in the ROM 132.If the booting is completed, the main CPU 133 copies various applicationprograms stored in the storage 145 to the RAM 131 and execute variousoperations by executing the application programs copied to the RAM 131.

The graphic processor 134 generates a screen including various objectssuch as icons, images, texts, and the like by using an operating unit(not shown) and a rendering unit (not shown). The operating unit (notshown) may calculate attribute values such as a coordinate value of eachobject to be displayed, a shape, a size, a color and the like thereofaccording to the layout of the screen by using the received controlcommand. The rendering unit (not shown) may generate screens havingvarious layouts including objects based on the attribute valuescalculated by the operating unit (not shown). The screen generated inthe rendering unit (not shown) may be displayed in a display region ofthe display 150.

The operation of the processor 130 described above may be performed byprograms stored in the storage 145.

The storage 145 may store various pieces of data such as an operationsystem (O/S) software module for operating the electronic device 100, anartificial intelligence module, an artificial intelligence dedicatedprogram module, an utterance history module, a user identificationmodule, a domain identification module, and a context identificationmodule, and the like.

The communicator 140 is a component which executes communication withvarious types of external devices according to various types ofcommunication systems. The communicator 140 includes a Wi-Fi chip 141, aBluetooth chip 142, a wireless communication chip 143, and an NFC chip144. The processor 130 executes communication with various externaldevices by using the communicator 140.

The Wi-Fi chip 141 and the Bluetooth chip 142 execute the communicationin a Wi-Fi system and a Bluetooth system, respectively. When using theWi-Fi chip 141 or the Bluetooth chip 142, various pieces of connectioninformation such as SSID or session key may be transmitted or receivedfirst to allow the communication connection by using these, and thenvarious pieces of information may be transmitted and received. Thewireless communication chip 143 means a chip executing the communicationbased on various communication standards such as IEEE, Zigbee, 3rdGeneration (3G), 3rd Generation Partnership Project (3GPP), and LongTerm Evolution (LTE). The NFC chip 144 means a chip operated in a nearfield communication (NFC) system using 13.56 MHz band among frequencybands of 135 kHz, 13.56 MHz, 433 MHz, 860 to 960 MHz, 2.45 GHz, and thelike.

In addition, the communicator 140 may further include a wiredcommunication interface such as HDMI, MHL, USB, DP, Thunderbolt, RGB,D-SUB, DVI, or the like. The processor 130 may be connected to theexternal server 200 via the wired communication interface of thecommunicator 140.

The display 150 may be implemented as various types of displays such asliquid crystal display (LCD), organic light emitting diodes (OLED)display, plasma display panel (PDP), and the like. The display 150 mayalso include a driving circuit or a backlight unit which may beimplemented in a form of a-si TFT, a low temperature poly silicon (LTPS)TFT, or an organic TFT (OTFT). The display 150 may be implemented as atouch screen by combining with a touch detector.

The user interface 155 receives various user interactions. The userinterface 155 may be implemented in various forms according to theimplementation example of the electronic device 100. For example, theuser interface 155 may be a button provided on the electronic device100, a microphone receiving a user's voice, or a camera detecting auser's motion. In addition, when the electronic device 100 isimplemented as a touch-based terminal device, the user interface 155 maybe implemented in a form of a touch screen forming a layer structurewith a touch pad. In this case, the user interface 155 may be used asthe display 150 described above.

The audio processor 170 is a component executing processing of audiodata. The audio processor 170 may execute various processing such asdecoding, amplification, or noise filtering regarding the audio data.

The video processor 180 is a component executing processing regardingvideo data. The video processor 180 may execute various image processingsuch as decoding, scaling, noise filtering, frame rate conversion, orresolution conversion regarding the video data.

The button 190 may be various types of buttons such as a mechanicalbutton, a touch pad, or a wheel formed in any region of a front portion,a side portion, or a rear portion of the appearance of the main body ofthe electronic device 100.

Through the method described above, the processor 130 may provide theresponse information with the maintained context, even if the user ischanged.

Hereinafter, the operations of the electronic device 100 will bespecifically described with reference to the drawings.

FIGS. 4A and 4B are views for explaining postponement of processing of avoice of a user among a plurality of users according to an embodiment ofthe disclosure.

Referring to FIG. 4A, when the first user's voice of “Add collegereunion in Gangnam in calendar” of the user A is input, the processor130 may identify the user A and provide the first response sound of“What date/what time is it?” to the first user's voice.

When the second user's voice of “Order Bulgogi Pizza” of the user B isinput, the processor 130 may identify the user B and store the seconduser's voice in the storage 145, because the identified user B isdifferent from the user A. The processor 130 may also store not only thesecond user's voice, but also the information regarding the user B whouttered the second user's voice in the storage 145.

When the third user's voice of “It's 7th of July” of the user A isinput, the processor 130 may identify the user A and provide the secondresponse sound of “Added on 17th of July” to the third user's voice,because the user is same as the user A who uttered the first user'svoice.

Referring to FIG. 4B, when it is determined that the interaction withthe user A has ended, the processor 130 may provide the third responsesound of “Sorry. I couldn't respond because I had dialogue task withfather yesterday. I put pizza you requested in shopping list. How manydo you want to order?” to the second user's voice. At this time, theprocessor 130 may provide the third response sound after detecting thatthe second user is nearby.

The processor 130 may determine that the interaction with the user A hasended, if the user A is not detected or a user's voice is not input fromthe user A for the predetermined period of time.

Later, when a fourth user's voice of “I understand. That's ok. Orderthree” of the user B is input, the processor 130 may identify the user Band provide a fourth response sound of “Yes. Ordered” to the fourthuser's voice, because it is the same as the user B who uttered thesecond user's voice.

FIGS. 5A to 5C are views for explaining a method for real-timesimultaneous processing of a plurality of user's voices according to anembodiment of the disclosure.

Referring to FIG. 5A, when the first user's voice of “When is wedding?”of the user A is input, the processor 130 may identify the user A andprovide the first response sound of “It's 7th of July” to the firstuser's voice.

Then, when the second user's voice of “How is the weather today?” of theuser B is input, the processor 130 may identify the user B and providethe second response sound of “Today's weather is cloudy” by consideringonly the second user's voice, regardless of the first utterance historyinformation including the first user's voice and the first responsesound, because the identified user B is different from the user A.

At this time, the processor 130 may provide the second response sound ina way different from the first response sound. For example, theprocessor 130 may provide the first response sound in a male voice andprovide the second response sound in a female voice.

In addition, when the identified user is a user stored in the storage145 in advance, the processor 130 may provide the response soundincluding a user name, ID, or the like. For example, if the user A isstored in the storage 145 in advance as ID of “Park” and the user B isstored in the storage 145 in advance as ID of “Kim”, the processor 130may provide the first response sound of “Park. It's 7th of July” and thesecond response sound of “Kim. Today's weather is cloudy”.

In addition, referring to FIGS. 5B and 5C, the processor 130 may controlthe display 150 to display colors different from each other according tothe target of the response sound to be provided. In addition, theelectronic device 100 may include an LED and control the LED to emitlight with colors different from each other according to the target ofthe response sound to be provided. Further, the electronic device 100may include a plurality of LEDs and the processor 130 may control theLED so that one of the plurality of LEDs is turned on according to thetarget of the response sound to be provided.

When the third user's voice of “Change it to 14th of July” of user A isinput, the processor 130 may identify the user A and provide the thirdresponse sound of “Changed” to the third user's voice based on the firstutterance history information, because the user is the same as the userA uttered the first user's voice.

Hereinabove, FIGS. 4A and 4B show the embodiment in which only oneuser's voice is processed although there are a plurality of user'svoices, and FIGS. 5A to 5C show the embodiment in which each input ofthe plurality of user's voices is separately processed. The electronicdevice 100 may provide a personal mode as shown in FIGS. 4A and 4B and acommon mode as shown in FIGS. 5A to 5C, and the processor 130 may beoperated in the personal mode or the common mode according to the modeselection of the user.

FIG. 6 is a view for explaining a collective interaction with aplurality of users according to an embodiment of the disclosure.

Referring to FIG. 6, when the first user's voice of “Add college reunionin Gangnam in calendar” of the user A is input, the processor 130 mayidentify the user A and provide the first response sound of “Whatdate/what time is it?” to the first user's voice.

When the second user's voice of “The college reunion is on Monday, 7thof July” of the user B is input, the processor 130 may identify the userB and provide the second response sound of “Added on 7th of July” byconsidering the first utterance history information including the firstuser's voice and the first response sound and the second user's voice,because although the identified user B is different from the user A, thecontext of the first user's voice is the same as the context of thesecond user's voice.

If it is not clear whether or not the context of the first user's voiceis the same as the context of the second user's voice, the processor 130may provide a response sound inquiring the users. For example, if it isnot clear whether or not the context of the first user's voice is thesame as the context of the second user's voice, the processor 130 mayprovide the third response sound of “Is the college reunion on Monday,7th of July, the college reunion in Gangnam?”.

When the contexts of the plurality of user's voices are changed whileexecuting the collective interaction as shown in FIG. 6, the processor130 may be operated in a common mode. In addition, when the contextsbetween the user's voices of the plurality of users coincide with eachother during the operation in the common mode, the processor 130 mayexecute the collective interaction.

FIG. 7 is a view for explaining an operation when a plurality of user'svoices respectively corresponding to a plurality of domains of the sameuser are input, according to an embodiment of the disclosure.

Referring to FIG. 7, when the first user's voice of “Turn the light offin another room when watching TV in the large room and make this roomcool and dark” of the user A is input and then the second user's voiceof “Add wedding schedule on next weekend” is immediately input, theprocessor 130 may identify the user A who uttered the first user's voiceand the second user's voice and may provide the first response sound of“What day, What time is it?” to the second user's voice which is inputlate. At this time, the processor 130 may store the first user's voicein the storage 145. In addition, the processor 130 may store not onlythe first user's voice, but also the information of the user A whouttered the first user's voice in the storage 145.

When the third user's voice of “Saturday, 12 o'clock” of the user A isinput, the processor 130 may identify the user A and provide the secondresponse sound of “OK. Added wedding schedule” to the third user's voicebased on the first utterance history information including the seconduser's voice and the first response sound, because the identified useris the same as the user A who uttered the first user's voice.

The processor 130 may determine that the interaction with the user hasended as the specific operation according to the user's voice iscompleted. In the above example, the processor 130 may determine thatthe first interaction with the user A has ended as the adding thewedding schedule is completed. However, there is no limitation thereto,and the processor 130 may also determine that the interaction with theuser has ended, if the user is not detected or the user's voice is notinput from the same user for a predetermined period of time or longer.In addition, the processor 130 may also determine that the interactionwith the user has ended, according to an end command of the user.

When it is determined that the interaction with the user has ended, theprocessor 130 may obtain the user's voice stored in the storage 145 andstart a new interaction. In the above example, the processor 130 mayobtain the first user's voice stored in the storage 145 and provide thethird response sound of “Do you want to turn off light of living roomand hallway from the request you made before?” to the first user'svoice.

At that time, if the user who uttered the user's voice stored in thestorage 145 is detected, the processor 130 may start a new interaction.In the above example, although the first user's voice stored in thestorage 145 is obtained, if the user A who uttered the first user'svoice is not detected, the processor 130 may not start a newinteraction.

When the fourth user's voice of “Turn off light of living room and turnon light of hallway” of the user A is input, the processor 130 mayprovide the fourth response sound of “OK” to the fourth user's voice.

FIGS. 8A and 8B are views for explaining a method for identifying a userwho uttered a user's voice according to an embodiment of the disclosure.

Referring to FIG. 8A, when the user's voice is input, the processor 130may identify the user from an image 810 captured by the camera 160. Inaddition, when the user's voice is input, the processor 130 may identifya user changing a shape of mouth from the image 810 captured by thecamera 160 as the user who uttered the user's voice.

Further, referring to FIG. 8B, the speaker 110 may include a pluralityof sub-speakers 110-1, 110-2, 110-3, and 110-4 installed to be spacedapart from each other, and the processor 130 may estimate a firstposition where the user's voice has uttered by using the plurality ofsub-speakers 110-1, 110-2, 110-3, and 110-4. When the additional user'svoice is input, the processor 130 may estimate a second position wherethe additional user's voice has uttered and identify the user as thesame user, if the second position is within a predetermined distancefrom the first position.

FIGS. 9A and 9B are views for explaining additional training of anartificial intelligence model according to an embodiment of thedisclosure.

As described above, the artificial intelligence model stored in theelectronic device 100 of the external server 200 may be trained inadvance by using a sample dialogue pattern or the like. Hereinafter, theartificial intelligence model may be retrained based on the interactionwith the user. The retraining means updating the artificial intelligencemodel trained with the sample dialogue pattern by additionally trainingthe artificial intelligence model based on the interaction with theuser.

Referring to FIG. 9A, when the first user's voice of “What's on TV now?”of the user A is input, the processor 130 may provide the first responsesound of “News 9” to the first user's voice.

When the second user's voice of “No movie?” of the user A is input, theprocessor 130 may execute the retraining of the artificial intelligencemodel based on the first user's voice, the first response sound, and thesecond user's voice.

For example, the retrained artificial intelligence model may be anartificial intelligence model obtained by reflecting a point that theuser prefers movie.

The same situation as in FIG. 9A is assumed in FIG. 9B. FIG. 9A is aview illustrating that the artificial intelligence model before theretraining is used and FIG. 9B is a view illustrating that the retrainedartificial intelligence model is used.

Referring to FIG. 9B, when the retrained artificial intelligence modelis used and the first user's voice of “What's on TV now? Of the user Ais input, the processor 130 may provide the second response sound of “Nomovie, TV series is on” to the first user's voice.

In FIGS. 9A and 9B, it is described that the electronic device 100stores the artificial intelligence model for convenience of description,but the above retraining method may be applied in the same manner to acase where the artificial intelligence model is stored in the externalserver 200.

FIG. 10 is a flowchart for explaining operations of the electronicdevice 100 storing the artificial intelligence model according to anembodiment of the disclosure.

The processor 130 receives a user's voice (S1010). The processor 130detects a user (S1020). The processor 130 determines whether or not theuser is the same as a previous user (S1030) and processes the user'svoice based on utterance history information, if the user is the sameuser (S1050).

If the user is not the same user, the processor determines whether ornot a domain is the same as a previous domain (S1040), and if the domainis the same, the processor 130 processes the user's voice based on theutterance history information (S1050). The processor 130 processes theuser's voice without considering the utterance history information, ifthe domain is not same (S1060).

The processor 130 provides the response sound to the voice processing(S1070).

FIG. 11 is a sequence diagram for explaining operations of theelectronic device 100 and the external server 200 storing the artificialintelligence model according to an embodiment of the disclosure.

The electronic device 100 receives the user's voice (S1110) and detectsthe user (S1120). The electronic device 100 transmits the userinformation and the user's voice to the external server 200.

The external server 200 generates response information corresponding tothe user's voice based on the user information (S1140). The externalserver 200 transmits the response information to the electronic device100 (S1150).

The electronic device 100 provides the response sound (S1160).

FIG. 12 is a block diagram illustrating a configuration of an externalserver 1300 for training and using an artificial intelligence modelwhich provides a response to a user's request according to an embodimentof the disclosure.

Referring to FIG. 12, the external server 1300 may include at least oneof a learning unit 1310 or a response unit 1320. The external server1300 of FIG. 12 may correspond to the external server 200 of FIG. 2.

The learning unit 1310 may generate or train an artificial intelligencemodel having a criterion for providing a response to the user's voice byusing learning data. The learning unit 1310 may generate a determinationmodel having a determination criterion by using collected learning data.

For example, the learning unit 1310 may perform training to obtaininformation of a voice and an image including a response to the user'svoice by using the user's voice as the learning data. In addition, thelearning unit 1310 may generate, train, or update the artificialintelligence model for providing information of a voice and an imageincluding the response to the user's voice by using the user's voice andstate information of the electronic device as the learning data.

The response unit 1320 may obtain information of a voice and an imageincluding a response to a predetermined user's voice by usingpredetermined data as input data of the trained artificial intelligencemodel.

For example, the response unit 1320 may obtain (or predict or infer) theinformation of the voice and the image including the response to theuser's voice by using the user's voice and the state information of theelectronic device as the input data of the trained artificialintelligence model.

In an embodiment of the disclosure, the learning unit 1310 and theresponse unit 1320 may be included in the external server 1300, but thisis merely an embodiment, and these may be mounted in the electronicdevice 100. Specifically, at least a part of the learning unit 1310 andat least a part of the response unit 1320 may be implemented as asoftware module or manufactured in a form of at least one hardware chipand mounted on the electronic device 100. For example, at least one ofthe learning unit 1310 or the response unit 1320 may be manufactured ina form of a dedicated hardware chip for artificial intelligence (AI) andmay be manufactured as a part of the conventional generic-purposeprocessor (e.g., CPU or application processor) or a graphic dedicatedprocessor (e.g., GPU) and mounted on various electronic devicesdescribed above. The dedicated hardware chip for artificial intelligenceis a dedicated processor specialized in possibility calculation and mayrapidly process the calculation operations in the artificialintelligence field such as machine learning with higher performance ofparallel processing than the conventional generic-purpose processor. Ifthe learning unit 1310 and the response unit 1320 are implemented assoftware module (or program module including instructions), the softwaremodule may be stored in a non-transitory computer readable medium. Inthis case, the software module may be provided by an operating system(O/S) or provided by a predetermined application. In addition, a part ofthe software module may be provided by the operating system (O/S) andthe other part thereof may be provided by the predetermined application.

In this case, the learning unit 1310 and the response unit 1320 may bemounted on one electronic device or may be mounted on separateelectronic devices, respectively. For example, one of the learning unit1310 and the response unit 1320 may be included in the electronic device100 and the other one thereof may be included in an external server. Inaddition, the learning unit 1310 and the response unit 1320 may beconnected in a wired or wireless manner so that the learning unit 1310may provide constructed model information to the response unit 1320 andthe data input to the response unit 1320 may be provided to the learningunit 1310 as additional learning data.

FIG. 13 is a block diagram illustrating the learning unit 1310 accordingto an embodiment of the disclosure.

Referring to FIG. 13, the learning unit 1310 according to an embodimentmay include a learning data obtaining unit 1310-1 and a model learningunit 1310-4. In addition, the learning unit 1310 may further selectivelyinclude at least one of a learning data preprocessing unit 1310-2, alearning data selection unit 1310-3, or a model evaluation unit 1310-5.

The learning data obtaining unit 1310-1 may obtain learning datanecessary for the artificial intelligence model for obtaining theresponse to the user's voice. In an embodiment of the disclosure, thelearning data obtaining unit 1310-1 may obtain the user's voice as thelearning data. In addition, the learning data obtaining unit 1310-1 mayobtain information necessary for obtaining a specific response to theuser's voice (e.g., state information of the electronic device) as thelearning data. The learning data may be data collected or tested by thelearning unit 1310 or a manufacturer of the learning unit 1310.

The model learning unit 1310-4 may train the artificial intelligencemodel to have a criterion to obtain the response to the user's voice byusing the learning data. For example, the model learning unit 1310-4 maytrain the artificial intelligence model through supervised learningusing at least a part of the learning data as the criterion to obtainthe response to the user's voice. In addition, the model learning unit1310-4 may train the artificial intelligence model, for example, throughunsupervised learning of finding out the criterion to provide theresponse by self-training using learning data without particularsupervision. In addition, the model learning unit 1310-4 may train theartificial intelligence model, for example, through reinforcementlearning using a feedback regarding whether or not a result of theresponse providing according to the training is accurate. Further, themodel learning unit 1310-4 may train the artificial intelligence model,for example, by using a learning algorithm including errorback-propagation or gradient descent.

The model learning unit 1310-4 may train a selectin criterion regardingwhich learning data is to be used to provide the information of thevoice and the image including the response to the user's voice by usingthe input data.

If the number of artificial intelligence models constructed in advanceis more than one, the model learning unit 1310-4 may determine anartificial intelligence model with basic learning data that is highlyrelevant to the input learning data as the artificial intelligence modelto be trained. In such a case, the basic learning data pieces may beclassified in advance for each type of data and the artificialintelligence model may be constructed in advance for each type of data.For example, the basic learning data pieces may be classified in advancebased on various criteria such as a region where the learning data isgenerated, time when the learning data is generated, a size of thelearning data, a genre of the learning data, a creator of the learningdata, a type of an object in the learning data, and the like.

When the artificial intelligence model is trained, the model learningunit 1310-4 may store the trained artificial intelligence model. In sucha case, the model learning unit 1310-4 may store the trained artificialintelligence model in a memory of the external server 1300. In addition,the model learning unit 1310-4 may store the trained artificialintelligence model in a memory of a server or an electronic deviceconnected to the external server 1300 via a wired or wireless network.

The learning unit 1310 may further include the learning datapreprocessing unit 1310-2 and the learning data selection unit 1310-3 inorder to improve a response result of the artificial intelligence modelor save resources or time necessary for providing the artificialintelligence model.

The learning data preprocessing unit 1310-2 may preprocess the obtaineddata so that the obtained data is used in the learning for obtaining theinformation including the response to the user's voice. The learningdata preprocessing unit 1310-2 may process the obtained data in apredefined format so that the obtained data is used for obtaining theinformation including the response to the user's voice. For example, thelearning data preprocessing unit 1310-2 may remove a text (e.g., adverb,exclamation, or the like) unnecessary when the artificial intelligencemodel provides the response, from the input user's voice.

The learning data selection unit 1310-3 may select data necessary forthe learning from data obtained by the learning data obtaining unit1310-1 or the data preprocessed by the learning data preprocessing unit1310-2. The selected learning data may be provided to the model learningunit 1310-4. The learning data selection unit 1310-3 may select learningdata necessary for learning from the obtained or preprocessed dataaccording to a predetermined selection criterion. In addition, thelearning data selection unit 1310-3 may select learning data accordingto the predetermined selection criterion according to the learningperformed by the model learning unit 1310-4.

The learning unit 1310 may further include the model evaluation unit1310-5 in order to improve a response result of the artificialintelligence model.

The model evaluation unit 1310-5 may input evaluation data to theartificial intelligence model and may cause the model learning unit1310-4 to train again, if the response result output from the evaluationdata does not satisfy the predetermined criterion. In such a case, theevaluation data may be predefined data for evaluating the artificialintelligence model.

For example, if a number or a rate of the evaluation data pieces havinginaccurate response results, among response results of the trainedartificial intelligence model regarding the evaluation data, exceeds apredetermined threshold value, the model evaluation unit 1310-5 mayevaluate that the predetermined criterion is not satisfied.

If the number of trained artificial intelligence models is more thanone, the model evaluation unit 1310-5 may evaluate whether or not eachof the trained artificial intelligence models satisfies thepredetermined criterion, and determine a model satisfying thepredetermined criterion as a final artificial intelligence model. Inthis case, if the number of models satisfying the predeterminedcriterion is more than one, the model evaluation unit 1310-5 maydetermine any one or a predetermined number of models set in advance inthe order of high evaluation grades as a final artificial intelligencemodel.

FIG. 14 is a block diagram illustrating the response unit 1320 accordingto an embodiment of the disclosure.

Referring to FIG. 14, the response unit 1320 according to an embodimentmay include an input data obtaining unit 1320-1 and a response resultproviding unit 1320-4.

In addition, the response unit 1320 may further selectively include atleast one of an input data preprocessing unit 1320-2, an input dataselection unit 1320-3, or a model updating unit 1320-5.

The input data obtaining unit 1320-1 may obtain data necessary forobtaining the response to the user's voice. The response resultproviding unit 1320-4 may obtain information of a voice and an imageincluding the response to the user's voice by applying the input dataobtained by the input data obtaining unit 1320-1 to an artificialintelligence model trained with an input value.

The response result providing unit 1320-4 may obtain a response resultby applying data selected by the input data preprocessing unit 1320-2 orthe input data selection unit 1320-3 to the artificial intelligencemodel with the input value. The response result may be determined by theartificial intelligence model.

In an embodiment, the response result providing unit 1320-4 may obtainthe information of the voice and the image including the response to theuser's voice by applying the user's voice obtained by the input dataobtaining unit 1320-1 and the electronic device to the trainedartificial intelligence model.

The response unit 1320 may further include the input data preprocessingunit 1320-2 and the input data selection unit 1320-3 in order to improvethe response result of the artificial intelligence model or saveresources or time necessary for providing the response result.

The input data preprocessing unit 1320-2 may preprocess the obtaineddata so that the obtained data is used in order to obtain the responseto the user's voice. The input data preprocessing unit 1320-2 mayprocess the obtained data in a predefined format so that the responseresult providing unit 1320-4 may use the obtained data to obtain theresponse to the user's voice.

The input data selection unit 1320-3 may select data necessary forproviding the response from data obtained by the input data obtainingunit 1320-1 or the data preprocessed by the input data preprocessingunit 1320-2. The selected data may be provided to the response resultproviding unit 1320-4. The input data selection unit 1320-3 may selectsome or all of the obtained or preprocessed data according to thepredetermined selection criterion for providing the response. Inaddition, the input data selection unit 1320-3 may select data accordingto the predetermined selection criterion by the training of the modellearning unit 1310-4.

The model updating unit 1320-5 may control to update the artificialintelligence model based on the evaluation regarding the response resultprovided by the response result providing unit 1320-4. For example, themodel updating unit 1320-5 may request so that the model learning unit1310-4 additionally trains or updates the artificial intelligence modelby providing the response result provided by the response resultproviding unit 1320-4 to the model learning unit 1310-4.

FIG. 15 is a view illustrating an example in which the electronic device100 and an external server S are linked to each other to train andidentify data according to an embodiment.

Referring to FIG. 15, the external server S may train the criterion forobtaining the information including the response to the user's voice,and the electronic device 100 may provide the response to the user'svoice with the voice and the image based on the learning result by theserver S.

In this case, the model learning unit 1310-4 of the server S may executea function of the learning unit 1310 illustrated in FIG. 13. That is,the model learning unit 1310-4 of the server S may train a criterionregarding which text information or state information of the electronicdevice is to be used for obtaining the response to the user's voice orhow to obtain the response to the user's voice by using the aboveinformation.

In addition, the response result providing unit 1320-4 of the electronicdevice 100 may obtain the response to the user's voice by applying thedata selected by the input data selection unit 1320-3 to the artificialintelligence model generated by the server S. Further, the responseresult providing unit 1320-4 of the electronic device 100 may receivethe artificial intelligence model generated by the server S to theserver S and obtain the response to the user's voice by using thereceived artificial intelligence model.

FIG. 16 is a flowchart for explaining a method for controlling anelectronic device according to an embodiment of the disclosure.

When a first user's voice is input, a user who uttered the first user'svoice is identified (S1610). A first response sound obtained byinputting the first user's voice to an artificial intelligence modeltrained with an artificial intelligence algorithm is provided (S1620).When a second user's voice is input, a user who uttered the seconduser's voice is identified (S1630). When the user who uttered the firstuser's voice is the same as the user who uttered the second user'svoice, a second response sound obtained by inputting the second user'svoice and utterance history information to the artificial intelligencemodel is provided (S1640).

The control method may further include, based on the user who utteredthe first user's voice being different from the user who uttered thesecond user's voice, storing a second user's voice, and based on auser's voice not being input from the user who uttered the first user'svoice for a predetermined period of time or longer, providing a thirdresponse sound obtained by inputting the second user's voice to theartificial intelligence model.

Step S1610 of identifying a user who uttered the first user's voice mayinclude, based on the first user's voice being input, identifying afirst user who uttered the first user's voice, Step S1620 of providing afirst response sound may include providing the first response soundobtained by inputting the first user's voice to the artificialintelligence model, step S1630 of identifying a user who uttered thesecond user's voice may include identifying a second user who utteredthe second user's voice, and the control method may further includeproviding a third response sound obtained by inputting the second user'svoice to the artificial intelligence model to be distinct from the firstresponse sound.

The control method may further include, based on a first additionaluser's voice being input by the first user's voice, providing a firstadditional response sound obtained by inputting the first additionaluser's voice and first utterance history information corresponding tothe first user's voice to the artificial intelligence model, and basedon a second additional user's voice being input by a second user,providing a second additional response sound obtained by inputting thesecond additional user's voice and second utterance history informationcorresponding to the second user's voice to the artificial intelligencemodel to be distinct from the first additional response sound.

The providing a first additional response sound may include displaying afirst UI corresponding to the first user while providing the firstadditional response sound, and the providing a second additionalresponse sound to be distinct from the first additional response soundmay include displaying a second UI corresponding to the second userwhile providing the second additional response sound.

The control method may further include, based on the user who utteredthe first user's voice not being the same as the user who uttered thesecond user's voice, obtaining a first domain corresponding to the firstuser's voice and a second domain corresponding to the second user'svoice, and based on the first domain being same as the second domain,providing a second response sound obtained by inputting the seconduser's voice and the utterance history information to the artificialintelligence model.

In the control method, when the second user's voice is input within afirst predetermined period of time from a point when the first user'svoice is input, the user who uttered the first user's voice is the sameas the user who uttered the second user's voice, and the first domaincorresponding to the first user's voice is different from the seconddomain corresponding to the second user's voice, the first user's voicemay be stored without providing the first response sound, and a thirdresponse sound obtained by inputting the second user's voice to theartificial intelligence model may be provided.

The control method may further include providing a first response soundobtained by inputting the first user's voice to the artificialintelligence model, when the user's voice corresponding to the seconddomain is not input for a second predetermined period of time from theuser.

Steps S1610 and S1630 of identifying a user may include identifying auser based on at least one of a captured image or an input user's voice.

Step S1610 of identifying a user who uttered the first user's voice mayinclude identifying a user who uttered the first user's voice based on ashape of mouth of the user included in an image for a period of timeduring which the first user's voice is input, and Step S1640 ofproviding a second response sound may include, based on the shape ofmouth of the identified user included in the image being changed for aperiod of time during which the second user's voice is input,identifying that the user who uttered the first user's voice is the sameas the user who uttered the second user's voice.

According to the embodiments of the disclosure described above, when theuser's voices are input from the plurality of users, respectively, theelectronic device may provide responses while maintaining the contextaccording to processing, postponing, or separately processing theplurality of user's voices based on the existing user's voice.

Hereinabove, it is described that one piece of utterance historyinformation is generated with respect to one user, but there is nolimitation thereto. For example, when the first user's voice of the userA is input in a state where the first utterance history information ofthe user A and the second utterance history information of the user Bare stored, the processor may provide a first response sound obtained byinputting the first user's voice and the first utterance historyinformation to the artificial intelligence model and a second responsesound obtained by inputting the first user's voice and the secondutterance history information to the artificial intelligence model.Through the operation described above, the processor may provide a senseof conversation with a plurality of utterers to the user.

The processor may obtain information such as a title, progresscompletion information, information in progress, time information, userinformation, final interaction information, keywords, content summary,and the like with respect to the utterance history information of thecompleted interaction and store these in the storage. Then, if the userutters a user's voice for searching for the past utterance historyinformation, the processor may obtain the utterance history informationcorresponding to the user's voice based on the information stored in thestorage. When an additional user's voice is input, the processor mayprovide a response sound to the additional user's voice based on theobtained utterance history information. That is, the user may executethe interaction to continue from the past utterance.

According to an embodiment of the disclosure, various embodiments of thedisclosure may be implemented as software including instructions storedin machine (e.g., computer)-readable storage media. The machine is adevice which invokes instructions stored in the storage medium and isoperated according to the invoked instructions, and may include anelectronic device (e.g., electronic device A) according to the disclosedembodiments. In a case where the instruction is executed by a processor,the processor may execute a function corresponding to the instructiondirectly or using other elements under the control of the processor. Theinstruction may include a code generated by a compiler or executed by aninterpreter. The machine-readable storage medium may be provided in aform of a non-transitory storage medium. Herein, the term“non-transitory” merely mean that the storage medium is tangible whilenot including signals, and it does not distinguish that data issemi-permanently or temporarily stored in the storage medium.

According to an embodiment of the disclosure, the methods according tovarious embodiments described above may be provided to be included in acomputer program product. The computer program product may be exchangedbetween a seller and a purchaser as a commercially available product.The computer program product may be distributed in the form of amachine-readable storage medium (e.g., compact disc read only memory(CD-ROM)) or distributed online through an application store (e.g.,PlayStore™). In a case of the on-line distribution, at least a part ofthe computer program product may be at least temporarily stored ortemporarily generated in a storage medium such as a memory of a serverof a manufacturer, a server of an application store, or a relay server.

According to an embodiment of the disclosure, the embodiments describedabove may be implemented in a recording medium readable by a computer ora similar device using software, hardware, or a combination thereof. Insome cases, the embodiments described in this specification may beimplemented as a processor itself. According to the implementation interms of software, the embodiments such as procedures and functionsdescribed in this specification may be implemented as separate softwaremodules. Each of the software modules may execute one or more functionsand operations described in this specification.

Computer instructions for executing processing operations according tothe embodiments descried above may be stored in a non-transitorycomputer-readable medium. When the computer instructions stored in sucha non-transitory computer-readable medium are executed by the processor,the computer instructions may enable a specific machine to execute theprocessing operations according to the embodiments described above.

The non-transitory computer-readable medium is not a medium storing datafor a short period of time such as a register, a cache, or a memory, butmeans a medium that semi-permanently stores data and is readable by amachine. Specific examples of the non-transitory computer-readablemedium may include a CD, a DVD, a hard disk, a Blu-ray disc, a USB, amemory card, and a ROM.

In addition, each of the elements (e.g., a module or a program)according to various embodiments described above may be composed of asingle entity or a plurality of entities, and some sub-elements of theabovementioned sub-elements may be omitted or other sub-elements may befurther included in various embodiments. Alternatively or additionally,some elements (e.g., modules or programs) may be integrated into oneentity to perform the same or similar functions performed by eachrespective element prior to integration. Operations performed by amodule, a program, or other elements, in accordance with variousembodiments, may be performed sequentially, in a parallel, repetitive,or heuristically manner, or at least some operations may be performed ina different order, omitted, or may add a different operation.

Hereinabove, the preferred embodiments of the disclosure have been shownand described, but the disclosure is not limited to specific embodimentsdescribed above, various modifications may be made by those skilled inthe art without departing from the gist of the disclosure claimed in theclaims, and such modifications may not be individually understood fromthe technical sprit or the prospect of the disclosure.

What is claimed is:
 1. An electronic device comprising: a microphonecomprising circuitry; a speaker comprising circuitry; and a processorelectrically connected to the microphone and the speaker, wherein theprocessor is configured to: based on a first user's voice being inputvia the microphone, identify a user who uttered the first user's voiceand provide a first response sound obtained by inputting the firstuser's voice to an artificial intelligence model trained through anartificial intelligence algorithm via the speaker, and based on a seconduser's voice being input via the microphone, identify a user who utteredthe second user's voice, and based on the user who uttered the firstuser's voice being the same as the user who uttered the second user'svoice, provide a second response sound obtained by inputting the seconduser's voice and utterance history information to the artificialintelligence model via the speaker.
 2. The device according to claim 1,further comprising: a storage comprising circuitry, wherein theprocessor is configured to: based on the user who uttered the seconduser's voice being different from the user who uttered the first user'svoice, store the second user's voice in the storage, and based on auser's voice not being input from the user who uttered the first user'svoice for a predetermined period of time or longer, provide a thirdresponse sound obtained by inputting the second user's voice to theartificial intelligence model via the speaker.
 3. The device accordingto claim 1, wherein the processor is further configured to: based on thefirst user's voice being input, identify a first user who uttered thefirst user's voice and provide the first response sound obtained byinputting the first user's voice to the artificial intelligence modelvia the speaker, and based on the second user's voice being input,identify a second user who uttered the second user's voice and provide athird response sound obtained by inputting the second user's voice tothe artificial intelligence model via the speaker to be distinct fromthe first response sound.
 4. The device according to claim 3, whereinthe processor is further configured to: based on a first additionaluser's voice being input by the first user, provide a first additionalresponse sound obtained by inputting the first additional user's voiceand first utterance history information corresponding to the firstuser's voice to the artificial intelligence model via the speaker, andbased on a second additional user's voice being input by the seconduser, provide a second additional response sound obtained by inputtingthe second additional user's voice and second utterance historyinformation corresponding to the second user's voice to the artificialintelligence model via the speaker to be distinct from the firstadditional response sound.
 5. The device according to claim 4, furthercomprising: a display electrically connected to the processor, whereinthe processor is further configured to: control the display to display afirst UI corresponding to the first user while providing the firstadditional response sound, and control the display to display a secondUI corresponding to the second user via the display while providing thesecond additional response sound.
 6. The device according to claim 1,wherein the processor is further configured to: based on the user whouttered the first user's voice being different from the user who utteredthe second user's voice, obtain a first domain corresponding to thefirst user's voice and a second domain corresponding to the seconduser's voice, and based on the first domain being the same as the seconddomain, provide the second response sound obtained by inputting thesecond user's voice and the utterance history information to theartificial intelligence model via the speaker.
 7. The device accordingto claim 1, further comprising: a storage electrically connected to theprocessor, wherein the processor is further configured to: based on thesecond user's voice being input within a first predetermined period oftime from a point when the first user's voice is input, the user whouttered the first user's voice being the same as the user who utteredthe second user's voice, and a first domain corresponding to the firstuser's voice being different from a second domain corresponding to thesecond user's voice, store the first user's voice in the storage withoutproviding the first response sound, and provide a third response soundobtained by inputting the second user's voice to the artificialintelligence model via the speaker.
 8. The device according to claim 7,wherein the processor is further configured to, based on a user's voicecorresponding to the second domain not being input from the user for asecond predetermined period of time or longer, provide the firstresponse sound obtained by inputting the first user's voice stored inthe storage to the artificial intelligence model via the speaker.
 9. Thedevice according to claim 1, further comprising: a camera comprisingcircuitry, wherein the processor is further configured to identify theuser based on at least one of an image captured by the camera or auser's voice input via the microphone.
 10. The device according to claim9, wherein the processor is further configured to: identify the user whouttered the first user's voice based on a shape of mouth of a userincluded in the image for a period of time during which the first user'svoice is input, and based on the shape of mouth of the identified userincluded in the image being changed for a period of time during whichthe second user's voice is input, identify the user who uttered thesecond user's voice to be the same as the user who uttered the firstuser's voice.
 11. A method for controlling an electronic device, themethod comprising: based on a first user's voice being input,identifying a user who uttered the first user's voice; providing a firstresponse sound obtained by inputting the first user's voice to anartificial intelligence model trained through an artificial intelligencealgorithm; based on a second user's voice being input, identifying auser who uttered the second user's voice; and based on the user whouttered the first user's voice being the same as the user who utteredthe second user's voice, providing a second response sound obtained byinputting the second user's voice and utterance history information tothe artificial intelligence model.
 12. The method according to claim 11,further comprising: based on the user who uttered the second user'svoice being different from the user who uttered the first user's voice,storing the second user's voice; and based on a user's voice not beinginput from the user who uttered the first user's voice for apredetermined period of time or longer, providing a third response soundobtained by inputting the second user's voice to the artificialintelligence model.
 13. The method according to claim 11, wherein theidentifying a user who uttered the first user's voice comprises, basedon the first user's voice being input, identifying a first user whouttered the first user's voice, wherein the providing a first responsesound comprises providing the first response sound obtained by inputtingthe first user's voice to the artificial intelligence model, wherein theidentifying a user who uttered the second user's voice comprisesidentifying a second user who uttered the second user's voice, andwherein the method further comprises providing a third response soundobtained by inputting the second user's voice to the artificialintelligence model to be distinct from the first response sound.
 14. Themethod according to claim 13, further comprising: based on a firstadditional user's voice being input by the first user, providing a firstadditional response sound obtained by inputting the first additionaluser's voice and first utterance history information corresponding tothe first user's voice to the artificial intelligence model; and basedon a second additional user's voice being input by the second user,providing a second additional response sound obtained by inputting thesecond additional user's voice and second utterance history informationcorresponding to the second user's voice to the artificial intelligencemodel to be distinct from the first additional response sound.
 15. Themethod according to claim 14, wherein the providing a first additionalresponse sound comprises a first UI corresponding to the first userwhile providing the first additional response sound, and wherein theproviding a second additional response sound to be distinct from thefirst additional response sound comprises displaying a second UIcorresponding to the second user while providing the second additionalresponse sound.